Camera peek into turn

ABSTRACT

Aspects of the disclosure relate to adjusting a virtual camera&#39;s orientation when a vehicle is making a turn. One or more computing devices may receive the vehicle&#39;s original heading prior to making the turn and the vehicle&#39;s current heading. Based on the vehicle&#39;s original heading and the vehicle&#39;s current heading, the one or more computing devices may determine an angle of a turn the vehicle is performing and The one or more computing devices may determine a camera rotation angle and adjust the virtual camera&#39;s orientation relative to the vehicle to an updated orientation by rotating the virtual camera by the camera rotation angle and generate a video corresponding to the virtual camera&#39;s updated orientation. The video may be displayed on the display by the one or more computing devices.

BACKGROUND

Autonomous vehicles, such as vehicles that do not require a humandriver, can be used to aid in the transport of passengers or items fromone location to another. Such vehicles may operate in a fully autonomousmode where passengers or a remote operator may provide some initialinput, such as a pickup or destination location, and the vehiclemaneuvers itself to that location.

Such vehicles are typically equipped with various types of sensors inorder to detect objects in the surroundings. For example, autonomousvehicles may include lasers, sonar, radar, cameras, and other deviceswhich scan and record data from the vehicle's surroundings. Sensor datafrom one or more of these devices may be used to detect objects andtheir respective characteristics (position, shape, heading, speed,etc.). These characteristics can be used to provide, to occupants of thevehicle, a visual indication of objects detected in the vicinity of avehicle as well as what those objects are likely to do for some briefperiod into the future.

To provide occupants of the vehicle with information about the vehicle'senvironment, a display in the vehicle may present a video generated by avirtual camera system from the perspective of a virtual camera fixedrelative to the vehicle at a default position. The video may includeimagery within a set field of view corresponding to the position of thevirtual camera including a portion of the vehicle's surroundings and thevehicle's projected trajectory. Upon the vehicle performing a turn, thevirtual camera will remain in the default position, such that the videowill continue to provide imagery located within the set field of view.

At the same time, such information can be used to control the vehicle inorder to avoid these objects, or minimize damage in the case of anunavoidable collision. Thus, detection, identification, and predictionare critical functions for the safe operation of autonomous vehicle.

BRIEF SUMMARY

This technology generally relates to adjusting a virtual camera'sorientation when a vehicle is making a turn. One or more computingdevices may receive the vehicle's original heading prior to making theturn and the vehicle's current heading. Based on the vehicle's originalheading and the vehicle's current heading, the one or more computingdevices may determine an angle of a turn the vehicle is performing. Theone or more computing devices may determine a camera rotation angle andadjust the virtual camera's orientation relative to the vehicle to anupdated orientation by rotating the virtual camera by the camerarotation angle. A video corresponding to the virtual camera's updatedorientation may be generated by the one or more computing devices anddisplayed on a display.

The vehicle's original heading and the vehicle's current heading may bereceived from the vehicle's positioning system.

Determining the angle of the turn may be determined by calculating thedifference between the vehicle's original heading from the vehicle'scurrent heading.

The camera rotation angle may be based on the angle of the turn, and iscalculated by inputting the angle of the turn into a squashing functionand receiving an outputted angle, wherein the outputted angle is thecamera rotation angle.

The virtual camera's orientation may be adjusted and the video may bedisplayed while the vehicle is making the turn.

The vehicle's current heading may be a projected heading and the virtualcamera's orientation may be updated before the vehicle performs theturn.

The camera rotation angle may be calculated by inputting the angle ofthe turn into a squashing function and receiving an outputted angle andthe outputted angle may be the camera rotation angle.

The camera rotation angle may be determined by accessing a tableincluding predetermined camera rotation angles for each angle of theturn.

A system for adjusting a virtual camera's orientation when a vehicle ismaking a turn may comprise one or more processors configured to receivethe vehicle's original heading prior to making the turn and thevehicle's current heading and determine, based on the vehicle's originalheading and the vehicle's current heading, an angle of a turn thevehicle is performing. The one or more processors may be furtherconfigured to determine a camera rotation angle, adjust the virtualcamera's orientation relative to the vehicle to an updated orientationby rotating the virtual camera by the camera rotation angle, generate avideo corresponding to the virtual camera's updated orientation, anddisplay the video.

A non-transitory computer readable medium on which instructions arestored, the instructions when executed by one or more processors, maycause the one or more processors to perform a method of adjusting avirtual camera's orientation when a vehicle is making a turn. The methodmay comprise receiving the vehicle's original heading prior to makingthe turn and the vehicle's current heading, determining, based on thevehicle's original heading and the vehicle's current heading, an angleof a turn the vehicle is performing, determining a camera rotationangle, adjusting the virtual camera's orientation relative to thevehicle to an updated orientation by rotating the virtual camera by thecamera rotation angle, generating a video corresponding to the virtualcamera's updated orientation, and displaying, on a display, the video.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional diagram of an example vehicle in accordance withaspects of the disclosure.

FIGS. 2A-2D are example external views of a vehicle in accordance withaspects of the disclosure.

FIG. 3 is a pictorial diagram of a system in accordance with aspects ofthe disclosure.

FIG. 4 is a functional diagram of the system of FIG. 3 in accordancewith aspects of the disclosure.

FIGS. 5A and 5B are illustrations of a virtual camera positioned at adefault position in accordance with aspects of the disclosure.

FIG. 6 is an illustration of a field of view of a virtual camera inaccordance with aspects of the disclosure.

FIG. 7 is an illustration of a field of view of a virtual camerarotating in accordance with aspects of the disclosure.

FIG. 8 is an illustration of a field of view of a virtual camera inbased on a vehicle's projected trajectory in accordance with aspects ofthe disclosure.

FIGS. 9-12 are illustrations of a field of view of a virtual camerachanging as a vehicle progresses through a turn, in accordance withaspects of the disclosure.

FIG. 13 is a chart showing the inputs and outputs of a squashingfunction in accordance with aspects of the disclosure.

FIGS. 14 and 15 are illustrations of a field of view of a virtual camerarotating in an opposite direction than the direction which a vehicle isprojected to turn in accordance with aspects of the disclosure.

FIG. 16 is an illustration of a field of view of a virtual camerapositioned above a vehicle in accordance with aspects of the disclosure.

FIG. 17 is a flow diagram in accordance with aspects of the disclosure.

DETAILED DESCRIPTION Overview

The technology generally relates to providing passengers withinformation about a vehicle's environment. This may include, forinstance, adjusting a virtual camera's position and orientation when thevehicle is making a turn. For instance, as the vehicle performs a turnthe position of the virtual camera may be rotated around the vehicle topresent video within an adjusted field of view corresponding to thevehicle's anticipated heading. In this regard, a computing device withinthe vehicle may transmit information to a virtual camera systemcorresponding to the vehicle's surroundings and positioning. Based onthe received information, the virtual camera system may generate thevideo from a perspective of the virtual camera using the received data.The video may be generated by overlaying the vehicle's projectedtrajectory and detected objects on a map corresponding to the route thevehicle is traveling.

The distance the virtual camera rotates around the vehicle may be basedon the angle of the turn the vehicle is performing. The virtual camerasystem may input the angle of the turn the vehicle is performing,relative to the heading the vehicle was traveling prior to the turn,into a rotation table and output a corresponding camera rotation angleby which the virtual camera may be rotated. The virtual camera systemmay continually update the camera rotation angle as the vehicleprogresses through the turn.

The virtual camera system may determine an anticipated angle of anupcoming turn of the vehicle based on the vehicle's projected trajectoryand begin to rotate the virtual camera at a predetermined distancebefore the vehicle makes the turn. For example, the virtual camerasystem may determine a distance the vehicle is from the upcoming turnand the anticipated angles of the turn the vehicle will be making. Upondetermining the projected turn is located at or less than thepredetermined distance, the virtual camera system may begin to rotatethe virtual camera through camera rotation angles corresponding to theanticipated angles of the turn.

The virtual camera may remain stationary when the vehicle is stopped orprojected to stop. For instance, the virtual camera may determine thevehicle's projected trajectory anticipates the vehicle will pass througha stop light. As such, the virtual camera may not rotate until the cartravels through the stop light.

To minimize minor camera rotations, such as those caused by minoroversteering corrections, and to smooth larger camera rotations, thevirtual camera system may use a squashing function. For instance, theangle of the turn the vehicle is performing may be input into thesquashing function and a camera angle rotation value, which is the same,or more gradual than the angles of the turn, may be output.

At unprotected turns, the virtual camera system may rotate the virtualcamera to generate imagery corresponding to oncoming traffic atunprotected turns. For instance, the virtual camera may rotate in theopposite direction of the vehicle's projected trajectory to generateimagery corresponding to other vehicles travelling on the road where thevehicle is attempting to turn.

The height and pitch of the virtual camera may also be adjusted tocapture more or less of the vehicle's surroundings. For example, whenthe vehicle is stopped at an intersection the virtual camera may bepositioned above the vehicle and the pitch of the virtual camera may beadjusted such that it looks straight down to the vehicle. The virtualcamera may then generate a video which includes imagery of the vehicle'ssurroundings on all sides of the vehicle.

The features described above may allow a vehicle, such as an autonomousvehicle, to provide video of the vehicle's trajectory and surroundingsto its passenger. By rotating the virtual camera generating the video,passengers are provided with information regarding the vehicle'sprojected trajectory or surroundings located further along the projectedtrajectory and consistent with the surroundings they would expect to seeif they were manually driving the vehicle. Further, the video of thevehicle's trajectory and surroundings may be provided to individuals orsystems located outside of the vehicle to the provide them with thevideo to remotely monitor and/or control the vehicle.

Example Systems

As shown in FIG. 1, a vehicle 100 in accordance with one aspect of thedisclosure includes various components. While certain aspects of thedisclosure are particularly useful in connection with specific types ofvehicles, the vehicle may be any type of vehicle including, but notlimited to, cars, trucks, motorcycles, busses, recreational vehicles,etc. The vehicle may have one or more computing devices, such ascomputing devices 110 containing one or more processors 120, memory 130and other components typically present in general purpose computingdevices.

The memory 130 stores information accessible by the one or moreprocessors 120, including instructions 132 and data 134 that may beexecuted or otherwise used by the processor 120. The memory 130 may beof any type capable of storing information accessible by the processor,including a computing device-readable medium, or other medium thatstores data that may be read with the aid of an electronic device, suchas a hard-drive, memory card, ROM, RAM, DVD or other optical disks, aswell as other write-capable and read-only memories. Systems and methodsmay include different combinations of the foregoing, whereby differentportions of the instructions and data are stored on different types ofmedia.

The instructions 132 may be any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor. For example, the instructions may be stored as computingdevice code on the computing device-readable medium. In that regard, theterms “instructions” and “programs” may be used interchangeably herein.The instructions may be stored in object code format for directprocessing by the processor, or in any other computing device languageincluding scripts or collections of independent source code modules thatare interpreted on demand or compiled in advance. Functions, methods androutines of the instructions are explained in more detail below.

The data 134 may be retrieved, stored or modified by processor 120 inaccordance with the instructions 132. For instance, although the claimedsubject matter is not limited by any particular data structure, the datamay be stored in computing device registers, in a relational database asa table having a plurality of different fields and records, XMLdocuments or flat files. The data may also be formatted in any computingdevice-readable format.

The one or more processor 120 may be any conventional processors, suchas commercially available CPUs. Alternatively, the one or moreprocessors may be a dedicated device such as an ASIC or otherhardware-based processor. Although FIG. 1 functionally illustrates theprocessor, memory, and other elements of computing devices 110 as beingwithin the same block, it will be understood by those of ordinary skillin the art that the processor, computing device, or memory may actuallyinclude multiple processors, computing devices, or memories that may ormay not be stored within the same physical housing. For example, memorymay be a hard drive or other storage media located in a housingdifferent from that of computing devices 110. Accordingly, references toa processor or computing device will be understood to include referencesto a collection of processors or computing devices or memories that mayor may not operate in parallel.

Computing devices 110 may include all of the components normally used inconnection with a computing device such as the processor and memorydescribed above as well as a user input 150 (e.g., a mouse, keyboard,touch screen and/or microphone) and various electronic displays (e.g., amonitor having a screen or any other electrical device that is operableto display information). In this example, the vehicle includes aninternal electronic display 152 as well as one or more speakers 154 toprovide information or audio visual experiences. In this regard,internal electronic display 152 may be located within a cabin of vehicle100 and may be used by computing devices 110 to provide information topassengers within the vehicle 100.

Computing devices 110 may also include one or more wireless networkconnections 156 to facilitate communication with other computingdevices, such as the client computing devices and server computingdevices described in detail below. The wireless network connections mayinclude short range communication protocols such as Bluetooth, Bluetoothlow energy (LE), cellular connections, as well as various configurationsand protocols including the Internet, World Wide Web, intranets, virtualprivate networks, wide area networks, local networks, private networksusing communication protocols proprietary to one or more companies,Ethernet, WiFi and HTTP, and various combinations of the foregoing.

In one example, computing devices 110 may be an autonomous drivingcomputing system incorporated into vehicle 100. The autonomous drivingcomputing system may be capable of communicating with various componentsof the vehicle. For example, returning to FIG. 1, computing devices 110may be in communication with various systems of vehicle 100, such asdeceleration system 160, acceleration system 162, steering system 164,signaling system 166, navigation system 168, positioning system 170,perception system 172, and virtual camera system 176 in order to controlthe movement, acceleration, speed, operation, etc. of vehicle 100 inaccordance with the instructions 132 of memory 130. Again, althoughthese systems are shown as external to computing devices 110, inactuality, these systems may also be incorporated into computing devices110, again as an autonomous driving computing system for controllingvehicle 100.

As an example, computing devices 110 may interact with decelerationsystem 160 and acceleration system 162 in order to control the speed andacceleration of the vehicle. For example, the acceleration system 162may provide signals to the engine 174 to accelerate at a particularrate. Similarly, steering system 164 may be used by computing devices110 in order to control the direction of vehicle 100. For example, ifvehicle 100 is configured for use on a road, such as a car or truck, thesteering system may include components to control the angle of wheels toturn the vehicle. Signaling system 166 may be used by computing devices110 in order to signal the vehicle's intent to other drivers orvehicles, for example, by lighting turn signals or brake lights whenneeded.

Navigation system 168 may be used by computing devices 110 in order todetermine and follow a route to a location. In this regard, thenavigation system 168 and/or data 134 may store detailed mapinformation, e.g., highly detailed maps identifying the shape andelevation of roadways, lane lines, intersections, crosswalks, speedlimits, traffic signals, buildings, signs, real time trafficinformation, vegetation, or other such objects and information. In otherwords, this detailed map information may define the geometry ofvehicle's expected environment including roadways as well as speedrestrictions (legal speed limits) for those roadways

Positioning system 170 may be used by computing devices 110 in order todetermine the vehicle's relative or absolute position on a map or on theearth. For example, the positioning system 170 may include a GPSreceiver to determine the device's latitude, longitude and/or altitudeposition. Other location systems such as laser-based localizationsystems, inertial-aided GPS, or camera-based localization may also beused to identify the location of the vehicle. The location of thevehicle may include an absolute geographical location, such as latitude,longitude, and altitude as well as relative location information, suchas location relative to other cars immediately around it which can oftenbe determined with less noise than absolute geographical location.

The positioning system 170 may also include other devices incommunication with computing devices 110, such as an accelerometer,gyroscope or another direction/speed detection device to determine thedirection and speed of the vehicle or changes thereto. By way of exampleonly, an acceleration device may determine its pitch, yaw or roll (orchanges thereto) relative to the direction of gravity or a planeperpendicular thereto. The device may also track increases or decreasesin speed and the direction of such changes. The device's provision oflocation and orientation data as set forth herein may be providedautomatically to the computing devices 110, other computing devices andcombinations of the foregoing.

The perception system 172 also includes one or more components fordetecting objects external to the vehicle such as other vehicles,obstacles in the roadway, traffic signals, signs, trees, etc. Forexample, the perception system 172 may include lasers, sonar, radar,cameras and/or any other detection devices that record data which may beprocessed by computing devices 110. In the case where the vehicle is asmall passenger vehicle such as a car, the car may include a laser orother sensors mounted on the roof or other convenient location. Forinstance, a vehicle's perception system may use various sensors, such asLIDAR, sonar, radar, cameras, etc. to detect objects and theircharacteristics such as location, orientation, size, shape, type,direction and speed of movement, etc. The raw data from the sensorsand/or the aforementioned characteristics can be quantified or arrangedinto a descriptive function or vector for processing by the computingdevices 110. As discussed in further detail below, computing devices 110may use the positioning system 170 to determine the vehicle's locationand perception system 172 to detect and respond to objects when neededto reach the location safely.

The virtual camera system 176 may aggregate data from the systems of thevehicle to generate a video of the vehicle's surroundings for display inthe vehicle. The video may include imagery within a set field of viewcorresponding to the position of the virtual camera including a portionof the vehicle's surroundings and the vehicle's projected trajectory.For instance, the virtual camera system may aggregate data from thevehicle's perception system, 172, such as objects external to thevehicle with a particular range, data from the positioning system 170,such as the vehicle's current location, and data from the navigationsystem 168, such as the vehicle's projected trajectory. The virtualcamera system may take such information and generate a video byoverlaying the vehicle's projected trajectory and detected objects on amap corresponding to the route the vehicle is traveling. The videogenerated by the virtual camera system 176 may be presented to thevehicle's passengers, other individuals, or recorded for future viewing,such as on a display in the vehicle. The virtual camera system mayinclude some or all of the components of the computing device 110 asdescribed herein, such as processor 120, memory 130, data 134,instructions 132, and wireless network connections 156. Likewise, inaddition to, or alternatively to the functions of the virtual camerasystem described herein, the virtual camera system may perform thefunctions of a computing device 110. In this regard, the virtual camerasystem may communicate with each system of the vehicle such as systems160-174.

FIGS. 2A-2D are examples of external views of vehicle 100. As can beseen, vehicle 100 includes many features of a typical vehicle such asheadlights 202, windshield 203, taillights/turn signal lights 204, rearwindshield 205, doors 206, side view mirrors 208, tires and wheels 210,and turn signal/parking lights 212. Headlights 202, taillights/turnsignal lights 204, and turn signal/parking lights 212 may be associatedthe signaling system 166. Light bar 207 may also be associated with thesignaling system 166.

Vehicle 100 also includes sensors of the perception system 172. Forexample, housing 214 may include one or more laser devices for having360 degree or narrower fields of view and one or more camera devices.Housings 216 and 218 may include, for example, one or more radar and/orsonar devices. The devices of the perception system 172 may also beincorporated into the typical vehicle components, such astaillights/turn signal lights 204 and/or side view mirrors 208. Each ofthese radar, camera, and lasers devices may be associated withprocessing components which process data from these devices as part ofthe perception system 172 and provide sensor data to the computingdevices 110.

The computing devices 110 may control the direction and speed of thevehicle by controlling various components. By way of example, computingdevices 110 may navigate the vehicle to a destination locationcompletely autonomously using data from the detailed map information andnavigation system 168. In order to maneuver the vehicle, computingdevices 110 may cause the vehicle to accelerate (e.g., by increasingfuel or other energy provided to the engine by acceleration system 162),decelerate (e.g., by decreasing the fuel supplied to the engine,changing gears, and/or by applying brakes by deceleration system 160),change direction (e.g., by turning the front or rear wheels of vehicle100 by steering system 164), and signal such changes (e.g., by lightingturn signals of signaling system 166). Thus, the acceleration system 162and deceleration system 160 may be a part of a drivetrain that includesvarious components between a power system 174 (for instance, a gas orelectric engine) of the vehicle and the wheels of the vehicle. Again, bycontrolling these systems, computing devices 110 may also control thedrivetrain of the vehicle in order to maneuver the vehicle autonomously.

The one or more computing devices 110 of vehicle 100 may also receive ortransfer information to and from other computing devices. FIGS. 3 and 4are pictorial and functional diagrams, respectively, of an examplesystem 300 that includes a plurality of computing devices 310, 320, and330 and a storage system 350 connected via a network 360. System 300also includes vehicle 100, and vehicle 100A which may be configuredsimilarly to vehicle 100. Although only a few vehicles and computingdevices are depicted for simplicity, a typical system may includesignificantly more computing devices, vehicles, and storage systems.

As shown in FIG. 3, each of computing devices 310, 320, and 330 mayinclude one or more processors, memory, data and instructions. Suchprocessors, memories, data and instructions may be configured similarlyto one or more processors 120, memory 130, data 134, and instructions132 of computing devices 110.

The network 360, and intervening nodes, may include variousconfigurations and protocols including short range communicationprotocols such as Bluetooth, Bluetooth LE, the Internet, World Wide Web,intranets, virtual private networks, wide area networks, local networks,private networks using communication protocols proprietary to one ormore companies, Ethernet, Wi-Fi and HTTP, and various combinations ofthe foregoing. Such communication may be facilitated by any devicecapable of transmitting data to and from other computing devices, suchas modems and wireless interfaces.

In one example, one or more computing devices 310 may include a serverhaving a plurality of computing devices, e.g., a load balanced serverfarm, that exchange information with different nodes of a network forthe purpose of receiving, processing and transmitting the data to andfrom other computing devices. For instance, one or more computingdevices 310 may include one or more server computing devices that arecapable of communicating with one or more computing devices 110 ofvehicle 100 or a similar computing device of vehicle 100A as well asclient computing devices 320 and 330 via the network 360. For example,vehicles 100 and 100A may be a part of a fleet of vehicles that can bemonitored and dispatched by server computing devices to variouslocations. In this regard, the vehicles of the fleet may periodicallysend the server computing devices location information provided by thevehicle's respective positioning systems and the one or more servercomputing devices may track the locations of the vehicles. Additionally,the vehicle's computing device 110 may provide video from the virtualcamera system to the one or more server computing devices for storageand/or future viewing.

In addition, server computing devices 310 may use network 360 totransmit and present information to a user, such as users 322 and 332 ona display, such as displays 324 and 334 of computing devices 320 and330. In this regard, computing devices 320 and 330 may be consideredclient computing devices.

As shown in FIG. 4, each client computing device 320 and 330 may be apersonal computing device intended for use by a user 322, 332, and haveall of the components normally used in connection with a personalcomputing device including a one or more processors (e.g., a centralprocessing unit (CPU)), memory (e.g., RAM and internal hard drives)storing data and instructions, a display such as displays 324, 334, 344(e.g., a monitor having a screen, a touch-screen, a projector, atelevision, or other device that is operable to display information),and user input devices 426, 436, 446 (e.g., a mouse, keyboard,touch-screen or microphone). The client computing devices may alsoinclude a camera for recording video streams, speakers, a networkinterface device, and all of the components used for connecting theseelements to one another.

Although the client computing devices 320 and 330 may each comprise afull-sized personal computing device, they may alternatively comprisemobile computing devices capable of wirelessly exchanging data with aserver over a network such as the Internet. By way of example only,client computing device 320 may be a mobile phone or a device such as awireless-enabled PDA, a tablet PC, a wearable computing device orsystem, laptop, or a netbook that is capable of obtaining informationvia the Internet or other networks. In another example, client computingdevice 330 may be a wearable computing device, such as a “smart watch.”As an example the user may input information using a keyboard, a keypad,a multi-function input button, a microphone, visual signals (forinstance, hand or other gestures) with a camera or other sensors, atouch screen, etc.

Storage system 350 may store various types of information and data, suchas videos generated by the virtual camera system 176. This informationmay be retrieved or otherwise accessed by a server computing device,such as one or more server computing devices 310, or client computingdevices 320 and 330, in order to perform some or all of the featuresdescribed herein. For example, the information may include user accountinformation such as credentials (e.g., a user name and password as inthe case of a traditional single-factor authentication as well as othertypes of credentials typically used in multi-factor authentications suchas random identifiers, biometrics, etc.) that can be used to identify auser to the one or more server computing devices. The user accountinformation may also include personal information such as the user'sname, contact information, identifying information of the user's clientcomputing device (or devices if multiple devices are used with the sameuser account), as well as one or more unique signals for the user.

As with memory 130, storage system 350 can be of any type ofcomputerized storage capable of storing information accessible by theserver computing devices 310, such as a hard-drive, memory card, ROM,RAM, DVD, CD-ROM, write-capable, and read-only memories. In addition,storage system 350 may include a distributed storage system where datais stored on a plurality of different storage devices which may bephysically located at the same or different geographic locations.Storage system 350 may be connected to the computing devices via thenetwork 360 as shown in FIG. 4 and/or may be directly connected to orincorporated into any of the computing devices 310, 320, 330, etc.

Example Methods

In addition to the operations described above and illustrated in thefigures, various operations will now be described. It should beunderstood that the following operations do not have to be performed inthe precise order described below. Rather, various steps can be handledin a different order or simultaneously, and steps may also be added oromitted.

A virtual camera system within the vehicle may generate video. In thisregard, the computing device may receive data from the vehicle'snavigation system, positioning system, and perception system. The datareceived from these systems may include a vehicle's projected trajectoryas well as data indicating objects, such as other vehicles, within thevicinity of the vehicle. For instance, projected trajectory informationmay be provided by the vehicle's navigation system 168 and positioningsystem 170 and the object data may be provided by the perception system172. The computing device may pass this data to the virtual camerasystem.

Based on the data received by the virtual camera system, a video may begenerated for display on a display in the vehicle for viewing by thevehicle's passengers. In this regard, the virtual camera system maygenerate the video from a perspective of a virtual camera using thereceived data by overlaying the vehicle's projected trajectory anddetected objects on a map corresponding to the route the vehicle istraveling. For instance, FIG. 6 shows one frame of a video generated bythe virtual camera system from the perspective of a virtual camerapositioned above and behind the vehicle. The video includes a map 601presented within a set field of view 503 of the virtual camera. Thevideo further includes a virtual representation of the vehicle 100 andthe vehicle's surroundings 610 overlaid on the map 601, along with thevehicle's projected trajectory 620. The video may include animations,illustrations, live-action, and/or other such content typically found invideos.

The perspective of a virtual camera may be fixed relative to the vehicleat a default position. For instance, this default position of the fixedposition virtual camera 501 may be above and to the rear of the vehicle100 and directed in a first heading 505, as shown in FIGS. 5A and 5B.The video may include imagery within a set field of view 503corresponding to the first heading 505 and position of the fixedposition virtual camera 501.

Upon the vehicle performing a turn, the fixed position virtual camera501 may remain in the default position relative to the vehicle, suchthat the video will continue to provide the vehicle's surroundings 610and the vehicle's projected trajectory 620 captured in the first heading505 and located within the set field of view 503 as shown in the examplevideo of FIG. 6.

As the vehicle performs a turn the position of the virtual camera may berotated around the vehicle to present video within an adjusted field ofview corresponding to the vehicle's anticipated heading. In this regard,the virtual camera may rotate a predetermined distance around thevehicle, from the default position, as the vehicle performs the turn.For example, upon the vehicle 100 making a turn in a first direction 710from a first road to a second road, the virtual camera 701 may rotatecounter clockwise from its fixed positioned to be directed at a newheading 705, as shown in FIG. 7. As such, the field of the view of thevirtual camera may correspond to an updated field of view 703.

The updated field of view 703 may be adjusted such that the videoincludes imagery of the second road 812 onto which the vehicle 100 isturning, as shown in FIG. 8. Although the above example shows thevehicle 100 turning from a first road 811 to a second road, as discussedbelow, the virtual camera rotation may occur when the vehicle 100 makesany turning motions, such as around a bend on a road or into a parkingspot.

The distance the virtual camera 701 rotates around the vehicle may bebased on the angle of the turn the vehicle is performing. The angle ofthe turn the vehicle is performing may be determined by measuring theangular difference between the original heading 705 which the vehiclewas originally on before performing a turn with the current heading 815which the vehicle is currently on, as further shown in FIG. 8. Theoriginal heading 705 and the current heading may be provided by thevehicle's navigation system 168 and positioning system 170.

The virtual camera system may input the angle of the turn the vehicle isperforming into a rotation table. The rotation table may map the angleof the turn to a camera rotation angle. For instance, the angle of theturn input into the rotation table may be 90° and the rotation table maymap the 90° angle to a 45° camera rotation. The virtual camera systemmay then rotate the virtual camera 701 by the camera rotation angle,thereby causing the field of view of the virtual camera 701 to alsorotate by the camera rotation angle.

As the vehicle progresses through the turn, the virtual camera systemmay continually update the camera rotation angle. For example, as shownin FIGS. 9-12, to complete a turn, the vehicle progresses through aseries of angles such as 0°-90°, or more or less.

The virtual camera system may continually adjust the camera rotationangle based on the angle of the turn the vehicle is currently performingrelative to the heading the vehicle was traveling prior to entering theturn. In this regard, the virtual camera system may update the camerarotation angle in substantially real-time by continually calculating acamera rotation angle based on the angle of the turn. For instance, thevehicle may be travelling along a projected trajectory 901 at an initialheading 903 and the virtual camera 701 may be positioned in the defaultposition with a first field of view 920.

As the vehicle 100 begins to turn, the heading of the vehicle may beupdated to a new heading 1003, as shown in FIG. 10. The differencebetween the initial heading and the new heading may be input into therotation table and a camera rotation angle may be output. The virtualcamera system may then rotate the virtual camera 701 by the outputcamera rotation angle, thereby causing the field of view of the virtualcamera 701 to also rotate by the camera rotation angle to show a secondfield of view 1020, as further shown in FIG. 10.

Similarly, as the vehicle 100 progresses along the turn, the differencebetween the initial heading 903 and the latest heading 1101, as shown inFIG. 11, may be input into the rotation table and a camera rotationangle may be output. The virtual camera system may then rotate thevirtual camera 701 by the output camera rotation angle, thereby causingthe field of view of the virtual camera 701 to also rotate by the camerarotation angle to show a third field of view 1120, as further shown inFIG. 11. The determination of the difference between the initial heading903 and the latest heading 1101 may occur every hundredth of a second,or more or less. Upon completion of the turn by the vehicle 100, thevirtual camera 701 may return back to the default position, therebycapturing field of view 1220 as shown in FIG. 12.

The virtual camera system may begin to rotate the virtual camera at apredetermined distance before the vehicle makes a turn. In this regardthe virtual camera system may determine an anticipated angle of anupcoming turn of the vehicle based on the vehicle's projectedtrajectory. For example, the virtual camera system may monitor thevehicle's projected trajectory and the positioning of the vehicle todetermine a distance the vehicle is from the upcoming turn and theanticipated angles of the turn the vehicle will be making. The virtualcamera system may compare how far away a projected turn is to apredetermined distance. Upon the virtual camera system determining theprojected turn is located at or less than the predetermined distance,the virtual camera system may begin to rotate the virtual camera throughcamera rotation angles corresponding to the anticipated angles of theturn.

The predetermined distance may be based on the speed of the vehicle andthe distance the vehicle is from the turn. In this regard, when thevehicle is traveling at a higher rate of speed the predetermineddistance may greater than when the vehicle is traveling at a slower rateof speed. This may allow for a smoother, more gradual rotation of thevirtual camera.

In some embodiments, the rotation of the virtual camera may temporarilycease when the vehicle is stopped or projected to stop. For instance,the virtual camera may receive the vehicle's projected trajectory anddetermine the vehicle is going to turn left but will first pass througha stop light. As such, the virtual camera may not rotate until the cartravels through the stop light.

A squashing function may be used to minimize minor camera rotations,such as those caused by minor oversteering corrections, etc., as well asto smooth larger camera rotations. In this regard, the squashingfunction may prevent the generated video from including imagery whichappears to constantly switch position or move too quickly. For instance,two example squashing results, 1310 and 1302 are further shown in FIG.13. To get squashing results, the angle of the turn the vehicle isperforming, as shown by the X-axis of FIG. 13, may be input into asquashing function. The squashing function may calculate and output acorresponding camera rotation angle, as shown in the Y-axis. The outputcamera angle rotation values, as shown on the Y-axis, which may be moreor less gradual than the angles of the turn, as shown on the X-axis.

As one example, the virtual camera system may rotate the virtual camerato generate imagery corresponding to oncoming traffic at unprotectedturns. In this regard, the virtual camera may rotate in the oppositedirection of the vehicle's projected trajectory 1430 to generate imagerycorresponding to other vehicles 1501 travelling on the road where thevehicle is attempting to turn 1420, as shown in FIGS. 14 and 15. Thecamera rotation angle may be predetermined for unprotected turns, suchas 45° or more or less. As such, the generated video may providepassengers of the vehicle with imagery similar to that which they wouldexperience were they driving the vehicle. Unprotected turns may includeturns where the vehicle is turning onto a road where traffic is moving,such as a right on red or right at a stop sign at an intersection wherethe road the car is turning onto does not require other vehicles to stopat the intersection.

The height and pitch of the virtual camera may also be adjusted tocapture more or less of the vehicle's surroundings. For example, whenthe vehicle is stopped at an intersection 1600 the virtual camera 710may be positioned above the vehicle 100 and the pitch of the virtualcamera may be adjusted such that it looks down to the vehicle, as shownin FIG. 16. The virtual camera 710 may look straight down on the vehicle100. The virtual camera 710 may then generate a video which includesimagery from all around the vehicle as further shown in FIG. 16.

FIG. 17 is an example flow diagram 1700 which may be performed by one ormore computing devices of a vehicle, such as computing device 110 ofvehicle 100 in order to adjust a virtual camera's orientation when avehicle is making a turn. In this example, the computing one or morecomputing devices may receive the vehicle's original heading prior tomaking the turn and the vehicle's current heading as shown in block1710. Based on the vehicle's original heading and the vehicle's currentheading, the one or more computing devices may determine an angle of aturn the vehicle is performing as shown in block 1720. The one or morecomputing devices may determine a camera rotation angle, as shown inblock 1730, and adjust the virtual camera's orientation relative to thevehicle to an updated orientation by rotating the virtual camera by thecamera rotation angle, as shown in block 1740. The one or more computingdevices may generate a video corresponding to the virtual camera'supdated orientation as shown in block 1750 and display the video on adisplay as shown in block 1760.

Unless otherwise stated, the foregoing alternative examples are notmutually exclusive, but may be implemented in various combinations toachieve unique advantages. As these and other variations andcombinations of the features discussed above can be utilized withoutdeparting from the subject matter defined by the claims, the foregoingdescription of the embodiments should be taken by way of illustrationrather than by way of limitation of the subject matter defined by theclaims. In addition, the provision of the examples described herein, aswell as clauses phrased as “such as,” “including” and the like, shouldnot be interpreted as limiting the subject matter of the claims to thespecific examples; rather, the examples are intended to illustrate onlyone of many possible embodiments. Further, the same reference numbers indifferent drawings can identify the same or similar elements.

1. A computer implemented method for adjusting a virtual camera'sorientation when a vehicle is making a turn, the method comprising:receiving, by one or more computing devices, the vehicle's originalheading prior to entering the turn and the vehicle's current heading;determining, based on the vehicle's original heading and the vehicle'scurrent heading, by the one or more computing devices, an angle of aturn the vehicle is performing; determining, by the one or morecomputing devices, a camera rotation angle; adjusting, by one or morecomputing devices, the virtual camera's orientation relative to thevehicle to an updated orientation by rotating the virtual camera by thecamera rotation angle; generating and displaying, by the one or morecomputing devices, a video corresponding to the virtual camera's updatedorientation.
 2. The method of claim 1, wherein the vehicle's originalheading and the vehicle's current heading are received from thevehicle's positioning system.
 3. The method of claim 1, whereindetermining the angle of the turn is determined by calculating thedifference between the vehicle's original heading from the vehicle'scurrent heading.
 4. The method of claim 1, wherein the camera rotationangle is based on the angle of the turn, and is calculated by inputtingthe angle of the turn into a squashing function and receiving anoutputted angle, wherein the outputted angle is the camera rotationangle.
 5. The method of claim 1, wherein the virtual camera'sorientation is adjusted and the video is displayed while the vehicle ismaking the turn.
 6. The method of claim 1, wherein the vehicle's currentheading is a projected heading and the virtual camera's orientation isupdated before the vehicle performs the turn.
 7. The method of claim 1,wherein the camera rotation angle is calculated by inputting the angleof the turn into a squashing function and receiving an outputted angle,wherein the outputted angle is the camera rotation angle.
 8. The methodof claim 1, wherein the camera rotation angle is determined by accessinga table including predetermined camera rotation angles for each angle ofthe turn.
 9. A system for adjusting a virtual camera's orientation whena vehicle is making a turn, the system comprising one or more processorsconfigured to: receive the vehicle's original heading prior to enteringthe turn and the vehicle's current heading; determine, based on thevehicle's original heading and the vehicle's current heading, an angleof a turn the vehicle is performing; determine a camera rotation angle;adjust the virtual camera's orientation relative to the vehicle to anupdated orientation by rotating the virtual camera by the camerarotation angle; generate and display a video corresponding to thevirtual camera's updated orientation.
 10. The system of claim 9, whereinthe vehicle's original heading and the vehicle's current heading arereceived from the vehicle's positioning system.
 11. The system of claim9, wherein determining the angle of the turn is determined bycalculating the difference between the vehicle's original heading fromthe vehicle's current heading.
 12. The system of claim 9, wherein thecamera rotation angle is based on the angle of the turn, and iscalculated by inputting the angle of the turn into a squashing functionand receiving an outputted angle, wherein the outputted angle is thecamera rotation angle.
 13. The system of claim 9, wherein the virtualcamera's orientation is adjusted and the video is displayed while thevehicle is making the turn.
 14. The system of claim 9, wherein thevehicle's current heading is a projected heading and the virtualcamera's orientation is updated before the vehicle performs the turn.15. The system of claim 9, wherein the camera rotation angle iscalculated by inputting the angle of the turn into a squashing functionand receiving an outputted angle, wherein the outputted angle is thecamera rotation angle.
 16. The method of claim 9, wherein the camerarotation angle is determined by accessing a table includingpredetermined camera rotation angles for each angle of the turn.
 17. Anon-transitory computer readable medium on which instructions arestored, the instructions when executed by one or more processors, causethe one or more processors to perform a method of adjusting a virtualcamera's orientation when a vehicle is making a turn, the methodcomprising: receiving the vehicle's original heading prior to enteringthe turn and the vehicle's current heading; determining, based on thevehicle's original heading and the vehicle's current heading, an angleof a turn the vehicle is performing; determining a camera rotationangle; adjusting the virtual camera's orientation relative to thevehicle to an updated orientation by rotating the virtual camera by thecamera rotation angle; generating and displaying a video correspondingto the virtual camera's updated orientation.
 18. The non-transitorycomputer readable medium of claim 17, wherein the vehicle's originalheading and the vehicle's current heading are received from thevehicle's positioning system.
 19. The non-transitory computer readablemedium of claim 17, wherein determining the angle of the turn isdetermined by calculating the difference between the vehicle's originalheading from the vehicle's current heading.
 20. The non-transitorycomputer readable medium of claim 17, wherein the camera rotation angleis based on the angle of the turn, and is calculated by inputting theangle of the turn into a squashing function and receiving an outputtedangle, wherein the outputted angle is the camera rotation angle.